Encrypted and watermarked temporal and resolution layering in advanced television

ABSTRACT

A method and apparatus for image compression using temporal and resolution layering of compressed image frames, and which provides encryption and watermarking capabilities. In particular, layered compression allows a form of modularized decomposition of an image that supports flexible encryption and watermarking techniques. Using layered compression, the base layer and various internal components of the base layer can be used to encrypt a compressed layered movie data stream. By using such a layered subset of the bits, the entire picture stream can be made unrecognizable by encrypting only a small fraction of the bits of the entire stream. A variety of encryption algorithms and strengths can be applied to various portions of the layered stream, including enhancement layers. Encryption algorithms or keys can be changed at each slice boundary as well, to provide greater intertwining of the encryption and the picture stream. Watermarking tracks lost or stolen copies back to the source, so that the nature of the method of theft can be determined and so that those involved in a theft can be identified. Watermarking preferably uses low order bits in certain coefficients in certain frames of a layered compression movie stream to provide reliable identification while being invisible or nearly invisible to the eye. An enhancement layer can also have its own unique identifying watermark structure.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part application of and claimspriority to U.S. application Ser. No. 09/442,595 filed on Nov. 17, 1999,now abnd., which was a continuation of U.S. application Ser. No.09/217,151 filed on Dec. 21, 1998, now U.S. Pat. No. 5,988,863, whichwas a continuation of U.S. application Ser. No. 08/594,815 filed Jan.30, 1996, (now U.S. Pat. No. 5,852,565, issued Dec. 22, 1998).

TECHNICAL FIELD

This invention relates to electronic communication systems, and moreparticularly to an advanced electronic television system having temporaland resolution layering of compressed image frames, and which providesencryption and watermarking capabilities.

BACKGROUND

The United States presently uses the NTSC standard for televisiontransmissions. However, proposals have been made to replace the NTSCstandard with an Advanced Television standard. For example, it has beenproposed that the U.S. adopt digital standard-definition and advancedtelevision formats at rates of 24 Hz, 30 Hz, 60 Hz, and 60 Hzinterlaced. It is apparent that these rates are intended to continue(and thus be compatible with) the existing NTSC television display rateof 60 Hz (or 59.94 Hz). It is also apparent that “3-2 pulldown” isintended for display on 60 Hz displays when presenting movies, whichhave a temporal rate of 24 frames per second (fps). However, while theabove proposal provides a menu of possible formats from which to select,each format only encodes and decodes a single resolution and frame rate.Because the display or motion rates of these formats are not integrallyrelated to each other, conversion from one to another is difficult.

Further, this proposal does not provide a crucial capability ofcompatibility with computer displays. These proposed image motion ratesare based upon historical rates which date back to the early part ofthis century. If a “clean-slate” were to be made, it is unlikely thatthese rates would be chosen. In the computer industry, where displayscould utilize any rate over the last decade, rates in the 70 to 80 Hzrange have proven optimal, with 72 and 75 Hz being the most commonrates. Unfortunately, the proposed rates of 30 and 60 Hz lack usefulinteroperability with 72 or 75 Hz, resulting in degraded temporalperformance.

In addition, it is being suggested by some in the field that frameinterlace is required, due to a claimed need to have about 1000 lines ofresolution at high frame rates, but based upon the notion that suchimages cannot be compressed within the available 18–19 mbits/second of aconventional 6 MHz broadcast television channel.

It would be much more desirable if a single signal format were to beadopted, containing within it all of the desired standard and highdefinition resolutions. However, to do so within the bandwidthconstraints of a conventional 6 MHz broadcast television channelrequires compression (or “scalability”) of both frame rate (temporal)and resolution (spatial). One method specifically intended to providefor such scalability is the MPEG-2 standard. Unfortunately, the temporaland spatial scalability features specified within the MPEG-2 standardare not sufficiently efficient to accommodate the needs of advancedtelevision for the U.S. Thus, the proposal for advanced television forthe U.S. is based upon the premise that temporal (frame rate) andspatial (resolution) layering are inefficient, and therefore discreteformats are necessary.

In addition to the above issues, the inventor has identified a need toprotect and manage the use of valuable copyrighted audio and video mediasuch as digital movies. The viability of entire technologies for moviedata delivery can hinge on the ability to protect and manage usage. Asthe quality of digital compressed movie masters approaches that of theoriginal work, the need for protection and management methodologiesbecomes a crucial requirement.

In approaching a system architecture for digital content protection andmanagement, it would be very beneficial to have a variety of tools andtechniques which can be applied in a modular and flexible way. Mostcommercial encryption systems have been eventually compromised. It istherefore necessary to architect any protection system to besufficiently flexible as to adapt and strengthen itself if and when itis compromised. It is also valuable to place informational clues intoeach copy via watermarking of symbols and/or serial number informationin order to pinpoint the source and method by which the security hasbeen compromised.

Movie distribution digitally to movie theaters is becoming feasible. Thehigh value copies of new movies have long been a target for theft orcopying of today's film prints. Digital media such as DVD have attemptedcrude encryption and authorization schemes (such as DIVX). Analog cablescramblers have been in use from the beginning to enable charging forpremium cable channels and pay-per-view events and movies. However thesecrude scramblers have been broadly compromised.

One reason that digital and analog video systems have tolerated suchpoor security systems is that the value of the secondary video releaseand the loss due to pirating is a relatively small proportion of themarket. However, for digital first-run movies, for valuable live events,and for high resolution images to the home and business (via forms ofHDTV), robust security systems become a requirement.

The present invention overcomes these and other problems of currentdigital content protection systems.

SUMMARY

The present invention provides a method and apparatus for imagecompression which demonstrably achieves better than 1000-line resolutionimage compression at high frame rates with high quality. It alsoachieves both temporal and resolution scalability at this resolution athigh frame rates within the available bandwidth of a conventionaltelevision broadcast channel. The inventive technique efficientlyachieves over twice the compression ratio being proposed for advancedtelevision while providing for flexible encryption and watermarkingtechniques.

Image material is preferably captured at an initial or primary framingrate of 72 fps. An MPEG-2 data stream is then generated, comprising:

-   (1) a base layer, preferably encoded using only MPEG-2 P frames,    comprising a low resolution (e.g., 1024×512 pixels), low frame rate    (24 or 36 Hz) bitstream;-   (2) an optional base resolution temporal enhancement layer, encoded    using only MPEG-2 B frames, comprising a low resolution (e.g.,    1024×512 pixels), high frame rate (72 Hz) bitstream;-   (3) an optional base temporal high resolution enhancement layer,    preferably encoded using only MPEG-2 P frames, comprising a high    resolution (e.g., 2k×1k pixels), low frame rate (24 or 36 Hz)    bitstream;-   (4) an optional high resolution temporal enhancement layer, encoded    using only MPEG-2 B frames, comprising a high resolution (e.g.,    2k×1k pixels), high frame rate (72 Hz) bitstream.

The invention provides a number of key technical attributes, allowingsubstantial improvement over current proposals, and including:replacement of numerous resolutions and frame rates with a singlelayered resolution and frame rate; no need for interlace in order toachieve better than 1000-lines of resolution for 2 megapixel images athigh frame rates (72 Hz) within a 6 MHz television channel;compatibility with computer displays through use of a primary framingrate of 72 fps; and greater robustness than the current unlayered formatproposal for advanced television, since all available bits may beallocated to a lower resolution base layer when “stressful” imagematerial is encountered.

The disclosed layered compression technology allows a form ofmodularized decomposition of an image. This modularity has additionalbenefits beyond allowing scalable decoding and better stress resilience.The modularity can be further tapped as a structure which supportsflexible encryption and watermarking techniques. The function ofencryption is to restrict viewing, performance, copying, or other use ofaudio/video shows unless one or more proper keys are applied to anauthorized decryption system. The function of watermarking is to tracklost or stolen copies back to a source, so that the nature of the methodof theft can be determined to improve the security of the system, and sothat those involved in the theft can be identified.

Using layered compression, the base layer, and various internalcomponents of the base layer (such as I frames and their DCcoefficients, or motion vectors for P frames) can be used to encrypt acompressed layered movie stream. By using such a layered subset of thebits, the entire picture stream can be made unrecognizable (unlessdecrypted) by encrypting only a small fraction of the bits of the entirepicture stream. Further, a variety of encryption algorithms andstrengths can be applied to various portions of the layered stream,including the enhancement layers (which can be seen as a premium qualityservice, and encrypted specially). Encryption algorithms or keys can bechanged at each slice boundary as well, to provide greater intertwiningof the encryption and the image stream.

The inventive layered compression structure can also be used forwatermarking. The goal of watermarking is to be reliably identifiable todetection, yet be essentially invisible to the eye. For example, loworder bits in DC coefficients in I frames would be invisible to the eye,but yet could be used to uniquely identify a particular picture streamwith a watermark. Enhancement layers can also have their own uniqueidentifying watermark structure.

The details of one or more embodiments of the invention are set forth inthe accompanying drawings and the description below. Other features,objects, and advantages of the invention will be apparent from thedescription and drawings, and from the claims.

DESCRIPTION OF DRAWINGS

FIG. 1 is a timing diagram showing the pulldown rates for 24 fps and 36fps material to be displayed at 60 Hz.

FIG. 2 is a first preferred MPEG-2 coding pattern.

FIG. 3 is a second preferred MPEG-2 coding pattern.

FIG. 4 is a block diagram showing temporal layer decoding in accordancewith the preferred embodiment of the present invention.

FIG. 5 is a block diagram showing 60 Hz interlaced input to a converterthat can output both 36 Hz and 72 Hz frames.

FIG. 6 is a diagram showing a “master template” for a base MPEG-2 layerat 24 or 36 Hz.

FIG. 7 is a diagram showing enhancement of a base resolution templateusing hierarchical resolution scalability utilizing MPEG-2.

FIG. 8 is a diagram showing the preferred layered resolution encodingprocess.

FIG. 9 is a diagram showing the preferred layered resolution decodingprocess.

FIG. 10 is a block diagram showing a combination of resolution andtemporal scalable options for a decoder in accordance with the presentinvention.

FIG. 11 is a diagram showing the scope of encryption and watermarking asa function of unit dependency.

FIGS. 12A and 12B show diagrams of image frames with different types ofwatermarks.

FIG. 13 is a flowchart showing one method of applying the encryptiontechniques of the invention.

FIG. 14 is a flowchart showing one method of applying the watermarkingtechniques of the invention.

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

Throughout this description, the preferred embodiment and examples shownshould be considered as exemplars, rather than as limitations on thepresent invention.

Temporal and Resolution Layering

Goals of a Temporal Rate Family

After considering the problems of the prior art, and in pursuing thepresent invention, the following goals were defined for specifying thetemporal characteristics of a future digital television system:

-   -   Optimal presentation of the high resolution legacy of 24        frame-per-second films.    -   Smooth motion capture for rapidly moving image types, such as        sports.    -   Smooth motion presentation of sports and similar images on        existing analog NTSC displays, as well as computer-compatible        displays operating at 72 or 75 Hz.    -   Reasonable but more efficient motion capture of        less-rapidly-moving images, such as news and live drama.    -   Reasonable presentation of all new digital types of images        through a converter box onto existing NTSC displays.    -   High quality presentation of all new digital types of images on        computer-compatible displays.    -   If 60 Hz digital standard or high resolution displays come into        the market, reasonable or high quality presentation on these        displays as well.

Since 60 Hz and 72/75 Hz displays are fundamentally incompatible at anyrate other than the movie rate of 24 Hz, the best situation would be ifeither 72/75 or 60 were eliminated as a display rate. Since 72 or 75 Hzis a required rate for N.I.I. (National Information Infrastructure) andcomputer applications, elimination of the 60 Hz rate as beingfundamentally obsolete would be the most future-looking. However, thereare many competing interests within the broadcasting and televisionequipment industries, and there is a strong demand that any new digitaltelevision infrastructure be based on 60 Hz (and 30 Hz). This has leadto much heated debate between the television, broadcast, and computerindustries.

Further, the insistence by some interests in the broadcast andtelevision industries on interlaced 60 Hz formats further widens the gapwith computer display requirements. Since non-interlaced display isrequired for computer-like applications of digital television systems, ade-interlacer is required when interlaced signals are displayed. Thereis substantial debate about the cost and quality of de-interlacers,since they would be needed in every such receiving device. Frame rateconversion, in addition to de-interlacing, further impacts cost andquality. For example, that NTSC to-from PAL converters continue to bevery costly and yet conversion performance is not dependable for manycommon types of scenes. Since the issue of interlace is a complex andproblematic subject, and in order to attempt to address the problems andissue of temporal rate, the invention is described in the context of adigital television standard without interlace.

Selecting Optimal Temporal Rates

Beat Problems. Optimal presentation on a 72 or 75 Hz display will occurif a camera or simulated image is created having a motion rate equal tothe display rate (72 or 75 Hz, respectively), and vice versa. Similarly,optimal motion fidelity on a 60 Hz display will result from a 60 Hzcamera or simulated image. Use of 72 Hz or 75 Hz generation rates with60 Hz displays results in a 12 Hz or 15 Hz beat frequency, respectively.This beat can be removed through motion analysis, but motion analysis isexpensive and inexact, often leading to visible artifacts and temporalaliasing. In the absence of motion analysis, the beat frequencydominates the perceived display rate, making the 12 or 15 Hz beat appearto provide less accurate motion than even 24 Hz. Thus, 24 Hz forms anatural temporal common denominator between 60 and 72 Hz. Although 75 Hzhas a slightly higher 15 Hz beat with 60 Hz, its motion is still not assmooth as 24 Hz, and there is no integral relationship between 75 Hz and24 Hz unless the 24 Hz rate is increased to 25 Hz. (In European 50 Hzcountries, movies are often played 4% fast at 25 Hz; this can be done tomake film presentable on 75 Hz displays.)

In the absence of motion analysis at each receiving device, 60 Hz motionon 72 or 75 Hz displays, and 75 or 72 Hz motion on 60 Hz displays, willbe less smooth than 24 Hz images. Thus, neither 72/75 Hz nor 60 Hzmotion is suitable for reaching a heterogeneous display populationcontaining both 72 or 75 Hz and 60 Hz displays.

3-2 Pulldown. A further complication in selecting an optimal frame rateoccurs due to the use of “3-2 pulldown” combined with video effectsduring the telecine (film-to-video) conversion process. During suchconversions, the 3-2 pulldown pattern repeats a first frame (or field) 3times, then the next frame 2 times, then the next frame 3 times, thenthe next frame 2 times, etc. This is how 24 fps film is presented ontelevision at 60 Hz (actually, 59.94 Hz for NTSC color). That is, eachof 12 pairs of 2 frames in one second of film is displayed 5 times,giving 60 images per second. The 3-2 pulldown pattern is shown in FIG.1.

By some estimates, more than half of all film on video has substantialportions where adjustments have been made at the 59.94 Hz video fieldrate to the 24 fps film. Such adjustments include “pan-and-scan”, colorcorrection, and title scrolling. Further, many films are time-adjustedby dropping frames or clipping the starts and ends of scenes to fitwithin a given broadcast scheduled. These operations can make the 3-2pulldown process impossible to reverse, since there is both 59.94 Hz and24 Hz motion. This can make the film very difficult to compress usingthe MPEG-2 standard. Fortunately, this problem is limited to existingNTSC-resolution material, since there is no significant library ofhigher resolution digital film using 3-2 pulldown.

Motion Blur. In order to further explore the issue of finding a commontemporal rate higher than 24 Hz, it is useful to mention motion blur inthe capture of moving images. Camera sensors and motion picture film areopen to sensing a moving image for a portion of the duration of eachframe. On motion picture cameras and many video cameras, the duration ofthis exposure is adjustable. Film cameras require a period of time toadvance the film, and are usually limited to being open only about 210out of 360 degrees, or a 58% duty cycle. On video cameras having CCDsensors, some portion of the frame time is often required to “read” theimage from the sensor. This can vary from 10% to 50% of the frame time.In some sensors, an electronic shutter must be used to blank the lightduring this readout time. Thus, the “duty cycle” of CCD sensors usuallyvaries from 50 to 90%, and is adjustable in some cameras. The lightshutter can sometimes be adjusted to further reduce the duty cycle, ifdesired. However, for both film and video, the most common sensor dutycycle duration is 50%.

Preferred Rate. With this issue in mind, one can consider the use ofonly some of the frames from an image sequence captured at 60, 72, or 75Hz. Utilizing one frame in two, three, four, etc., the subrates shown inTABLE 1 can be derived.

TABLE 1 Rate 1/2 Rate 1/3 Rate 1/4 Rate 1/5 Rate 1/6 Rate 75 Hz 37.5 2518.25 15 12.5 72 Hz 36 24 18 14.4 12 60 Hz 30 20 15 12 10

The rate of 15 Hz is a unifying rate between 60 and 75 Hz. The rate of12 Hz is a unifying rate between 60 and 72 Hz. However, the desire for arate above 24 Hz eliminates these rates. 24 Hz is not common, but theuse of 3-2 pulldown has come to be accepted by the industry forpresentation on 60 Hz displays. The only candidate rates are therefore30, 36, and 37.5 Hz. Since 30 Hz has a 7.5 Hz beat with 75 Hz, and a 6Hz beat with 72 Hz, it is not useful as a candidate.

The motion rates of 36 and 37.5 Hz become prime candidates for smoothermotion than 24 Hz material when presented on 60 and 72/75 Hz displays.Both of these rates are about 50% faster and smoother than 24 Hz. Therate of 37.5 Hz is not suitable for use with either 60 or 72 Hz, so itmust be eliminated, leaving only 36 Hz as having the desired temporalrate characteristics. (The motion rate of 37.5 Hz could be used if the60 Hz display rate for television can be move 4% to 62.5 Hz. Given theinterests behind 60 Hz, 62.5 Hz appears unlikely—there are even thosewho propose the very obsolete 59.94 Hz rate for new television systems.However, if such a change were to be made, the other aspects of thepresent invention could be applied to the 37.5 Hz rate.)

The rates of 24, 36, 60, and 72 Hz are left as candidates for a temporalrate family. The rates of 72 and 60 Hz cannot be used for a distributionrate, since motion is less smooth when converting between these tworates than if 24 Hz is used as the distribution rate, as describedabove. By hypothesis, we are looking for a rate faster than 24 Hz.Therefore, 36 Hz is the prime candidate for a master, unifying motioncapture and image distribution rate for use with 60 and 72/75 Hzdisplays.

As noted above, the 3-2 pulldown pattern for 24 Hz material repeats afirst frame (or field) 3 times, then the next frame 2 times, then thenext frame 3 times, then the next frame 2 times, etc. When using 36 Hz,each pattern optimally should be repeated in a 2-1-2 pattern. This canbe seen in TABLE 2 and graphically in FIG. 1.

TABLE 2 Rate Frame Numbers 60 Hz 1 2 3 4 5 6 7 8 9 10 24 Hz 1 1 1 2 2 33 3 4 4 36 Hz 1 1 2 3 3 4 4 5 6 6

This relationship between 36 Hz and 60 Hz only holds for true 36 Hzmaterial. 60 Hz material can be “stored” in 36 Hz, if it is interlaced,but 36 Hz cannot be reasonable created from 60 Hz without motionanalysis and reconstruction. However, in looking for a new rate formotion capture, 36 Hz provides slightly smoother motion on 60 Hz thandoes 24 Hz, and provides substantially better image motion smoothness ona 72 Hz display. Therefore, 36 Hz is the optimum rate for a master,unifying motion capture and image distribution rate for use with 60 and72 Hz displays, yielding smoother motion than 24 Hz material presentedon such displays.

Although 36 Hz meets the goals set forth above, it is not the onlysuitable capture rate. Since 36 Hz cannot be simply extracted from 60Hz, 60 Hz does not provide a suitable rate for capture. However, 72 Hzcan be used for capture, with every other frame then used as the basisfor 36 Hz distribution. The motion blur from using every other frame of72 Hz material will be half of the motion blur at 36 Hz capture. Testsof motion blur appearance of every third frame from 72 Hz show thatstaccato strobing at 24 Hz is objectionable. However, utilizing everyother frame from 72 Hz for 36 Hz display is not objectionable to the eyecompared to 36 Hz native capture.

Thus, 36 Hz affords the opportunity to provide very smooth motion on 72Hz displays by capturing at 72 Hz, while providing better motion on 60Hz displays than 24 Hz material by using alternate frames of 72 Hznative capture material to achieve a 36 Hz distribution rate and thenusing 2-1-2 pulldown to derive a 60 Hz image.

In summary, TABLE 3 shows the preferred optimal temporal rates forcapture and distribution in accordance with the present invention.

TABLE 3 Preferred Rates Capture Distribution Optimal Display AcceptableDisplay 72 Hz 36 Hz + 36 Hz 72 Hz 60 Hz

It is also worth noting that this technique of utilizing alternateframes from a 72 Hz camera to achieve a 36 Hz distribution rate canprofit from an increased motion blur duty cycle. The normal 50% dutycycle at 72 Hz, yielding a 25% duty cycle at 36 Hz, has beendemonstrated to be acceptable, and represents a significant improvementover 24 Hz on 60 Hz and 72 Hz displays. However, if the duty cycle isincreased to be in the 75–90% range, then the 36 Hz samples would beginto approach the more common 50% duty cycle. Increasing the duty rate maybe accomplished, for example, by using “backing store” CCD designs whichhave a short blanking time, yielding a high duty cycle. Other methodsmay be used, including dual CCD multiplexed designs.

Modified MPEG-2 Compression

For efficient storage and distribution, digital source material havingthe preferred temporal rate of 36 Hz should be compressed. The preferredform of compression for the present invention is accomplished by using anovel variation of the MPEG-2 standard.

MPEG-2 Basics. MPEG-2 is an international video compression standarddefining a video syntax that provides an efficient way to representimage sequences in the form of more compact coded data. The language ofthe coded bits is the “syntax.” For example, a few tokens can representan entire block of 64 samples. MPEG also describes a decoding(reconstruction) process where the coded bits are mapped from thecompact representation into the original, “raw” format of the imagesequence. For example, a flag in the coded bitstream signals whether thefollowing bits are to be decoded with a discrete cosine transform (DCT)algorithm or with a prediction algorithm. The algorithms comprising thedecoding process are regulated by the semantics defined by MPEG. Thissyntax can be applied to exploit common video characteristics such asspatial redundancy, temporal redundancy, uniform motion, spatialmasking, etc. In effect, MPEG-2 defines a programming language as wellas a data format. An MPEG-2 decoder must be able to parse and decode anincoming data stream, but so long as the data stream complies with theMPEG-2 syntax, a wide variety of possible data structures andcompression techniques can be used. The present invention takesadvantage of this flexibility by devising a novel means and method fortemporal and resolution scaling using the MPEG-2 standard.

MPEG-2 uses an intraframe and an interframe method of compression. Inmost video scenes, the background* remains relatively stable whileaction takes place in the foreground. The background may move, but agreat deal of the scene is redundant. MPEG-2 starts its compression bycreating a reference frame called an I (for Intra) frame. I frames arecompressed without reference to other frames and thus contain an entireframe of video information. I frames provide entry points into a databitstream for random access, but can only be moderately compressed.Typically, the data representing I frames is placed in the bitstreamevery 10 to 15 frames. Thereafter, since only a small portion of theframes that fall between the reference I frames are different from thebracketing I frames, only the differences are captured, compressed andstored. Two type of frames are used for such differences—P (forPredicted) frames and B (for Bi-directional Interpolated) frames.

P frames generally are encoded with reference to a past frame (either anI frame or a previous P frame), and, in general, will be used as areference for future P frames. P frames receive a fairly high amount ofcompression. B frames pictures provide the highest amount of compressionbut generally require both a past and a future reference in order to beencoded. Bi-directional frames are never used for reference frames.

Macroblocks within P frames may also be individually encoded usingintra-frame coding. Macroblocks within B frames may also be individuallyencoded using intra-frame coding, forward predicted coding, backwardpredicted coding, or both forward and backward, or bi-directionallyinterpolated, predicted coding. A macroblock is a 16×16 pixel groupingof four 8×8 DCT blocks, together with one motion vector for P frames,and one or two motion vectors for B frames.

After coding, an MPEG data bitstream comprises a sequence of I, P, and Bframes. A sequence may consist of almost any pattern of I, P, and Bframes (there are a few minor semantic restrictions on their placement).However, it is common in industrial practice to have a fixed pattern(e.g., IBBPBBPBBPBBPBB).

As an important part of the present invention, an MPEG-2 data stream iscreated comprising a base layer, at least one optional temporalenhancement layer, and an optional resolution enhancement layer. Each ofthese layers will be described in detail.

Temporal Scalability

Base Layer. The base layer is used to carry 36 Hz source material. Inthe preferred embodiment, one of two MPEG-2 frame sequences can be usedfor the base layer: IBPBPBP or IPPPPPP. The latter pattern is mostpreferred, since the decoder would only need to decode P frames,reducing the required memory bandwidth if 24 Hz movies were also decodedwithout B frames.

72 Hz Temporal Enhancement Layer. When using MPEG-2 compression, it ispossible to embed a 36 Hz temporal enhancement layer as B frames withinthe MPEG-2 sequence for the 36 Hz base layer if the P frame distance iseven. This allows the single data stream to support both 36 Hz displayand 72 Hz display. For example, both layers could be decoded to generatea 72 Hz signal for computer monitors, while only the base layer might bedecoded and converted to generate a 60 Hz signal for television.

In the preferred embodiment, the MPEG-2 coding patterns ofIPBBBPBBBPBBBP or IPBPBPBPB both allow placing alternate frames in aseparate stream containing only temporal enhancement B frames to take 36Hz to 72 Hz. These coding patterns are shown in FIGS. 2 and 3,respectively. The 2-Frame P spacing coding pattern of FIG. 3 has theadded advantage that the 36 Hz decoder would only need to decode Pframes, reducing the required memory bandwidth if 24 Hz movies were alsodecoded without B frames.

Experiments with high resolution images have suggested that the 2-FrameP spacing of FIG. 3 is optimal for most types of images. That is, theconstruction in FIG. 3 appears to offer the optimal temporal structurefor supporting both 60 and 72 Hz, while providing excellent results onmodern 72 Hz computer-compatible displays. This construction allows twodigital streams, one at 36 Hz for the base layer, and one at 36 Hz forthe enhancement layer B frames to achieve 72 Hz. This is illustrated inFIG. 4, which is a block diagram showing that a 36 Hz base layer MPEG-2decoder 50 simply decodes the P frames to generate 36 Hz output, whichmay then be readily converted to either 60 Hz or 72 Hz display. Anoptional second decoder 52 simply decodes the B frames to generate asecond 36 Hz output, which when combined with the 36 Hz output of thebase layer decoder 50 results in a 72 Hz output (a method for combiningis discussed below). In an alternative embodiment, one fast MPEG-2decoder 50 could decode both the P frames for the base layer and the Bframes for the enhancement layer.

Optimal Master Format. A number of companies are building MPEG-2decoding chips which operate at around 11 MPixels/second. The MPEG-2standard has defined some “profiles” for resolutions and frame rates.Although these profiles are strongly biased toward computer-incompatibleformat parameters such as 60 Hz, non-square pixels, and interlace, manychip manufacturers appear to be developing decoder chips which operateat the “main profile, main level”. This profile is defined to be anyhorizontal resolution up to 720 pixels, any vertical resolution up to576 lines at up to 25 Hz, and any frame rate of up to 480 lines at up to30 Hz. A wide range of data rates from approximately 1.5 Mbits/second toabout 10 Mbits/second is also specified. However, from a chip point ofview, the main issue is the rate at which pixels are decoded. Themain-level, main-profile pixel rate is about 10.5 MPixels/second.

Although there is variation among chip manufacturers, most MPEG-2decoder chips will in fact operate at up to 13 MPixels/second, givenfast support memory. Some decoder chips will go as fast as 20MPixels/second or more. Given that CPU chips tend to gain 50%improvement or more each year at a given cost, one can expect somenear-term flexibility in the pixel rate of MPEG-2 decoder chips.

TABLE 4 illustrates some desirable resolutions and frame rates, andtheir corresponding pixel rates.

TABLE 4 Resolution X Y Frame Rate (Hz) Pixel Rate (MPixels/s) 640 480 3611.1 720 486 36 12.6 720 486 30 (for comparison) 10.5 704 480 36 12.2704 480 30 (for comparison) 10.1 680 512 36 12.5 1024 512 24 12.6

All of these formats can be utilized with MPEG-2 decoder chips that cangenerate at least 12.6 MPixels/second. The very desirable 640×480 at 36Hz format can be achieved by nearly all current chips, since its rate is11.1 MPixels/second. A widescreen 1024×512 image can be squeezed into680×512 using a 1.5:1 squeeze, and can be supported at 36 Hz if 12.5MPixels/second can be handled. The highly desirable square pixelwidescreen template of 1024×512 can achieve 36 Hz when MPEG-2 decoderchips can process about 18.9 MPixels/second. This becomes more feasibleif 24 Hz and 36 Hz material is coded only with P frames, such that Bframes are only required in the 72 Hz temporal enhancement layerdecoders. Decoders which use only P frames require less memory andmemory bandwidth, making the goal of 19 MPixels/second more accessible.

The 1024×512 resolution template would most often be used with 2.35:1and 1.85:1 aspect ratio films at 24 fps. This material only requires11.8 MPixels/second, which should fit within the limits of most existingmain level-main profile decoders.

All of these formats are shown in FIG. 6 in a “master template” for abase layer at 24 or 36 Hz. Accordingly, the present invention provides aunique way of accommodating a wide variety of aspect ratios and temporalresolution compared to the prior art. (Further discussion of a mastertemplate is set forth below).

The temporal enhancement layer of B frames to generate 72 Hz can bedecoded using a chip with double the pixel rates specified above, or byusing a second chip in parallel with additional access to the decodermemory. Under the present invention, at least two ways exist for mergingof the enhancement and base layer data streams to insert the alternate Bframes. First, merging can be done invisibly to the decoder chip usingthe MPEG-2 transport layer. The MPEG-2 transport packets for two PIDs(Program IDs) can be recognized as containing the base layer andenhancement layer, and their stream contents can both be simply passedon to a double-rate capable decoder chip, or to an appropriatelyconfigured pair of normal rate decoders. Second, it is also possible touse the “data partitioning” feature in the MPEG-2 data stream instead ofthe transport layer from MPEG-2 systems. The data partitioning featureallows the B frames to be marked as belonging to a different classwithin the MPEG-2 compressed data stream, and can therefore be flaggedto be ignored by 36-Hz decoders which only support the temporal baselayer rate.

Temporal scalability, as defined by MPEG-2 video compression, is not asoptimal as the simple B frame partitioning of the present invention. TheMPEG-2 temporal scalability is only forward referenced from a previous Por B frame, and thus lacks the efficiency available in the B frameencoding proposed here, which is both forward and backward referenced.Accordingly, the simple use of B frames as a temporal enhancement layerprovides a simpler and more efficient temporal scalability than does thetemporal scalability defined within MPEG-2. Notwithstanding, this use ofB frames as the mechanism for temporal scalability is fully compliantwith MPEG-2. The two methods of identifying these B frames as anenhancement layer, via data partitioning or alternate PID's for the Bframes, are also fully compliant.

50/60 Hz Temporal enhancement layer. In addition to, or as analternative to, the 72 Hz temporal enhancement layer described above(which encodes a 36 Hz signal), a 60 Hz temporal enhancement layer(which encodes a 24 Hz signal) can be added in similar fashion to the 36Hz base layer. A 60 Hz temporal enhancement layer is particular usefulfor encoding existing 60 Hz interlaced video material.

Most existing 60 Hz interlaced material is video tape for NTSC inanalog, D1, or D2 format. There is also a small amount of Japanese HDTV(SMPTE 240/260M). There are also cameras which operate in this format.Any such 60 Hz interlaced format can be processed in known fashion suchthat the signal is de-interlaced and frame rate converted. This processinvolves very complex image understanding technology, similar to robotvision. Even with very sophisticated technology, temporal aliasinggenerally will result in “misunderstandings” by the algorithm andoccasionally yield artifacts. Note that the typical 50% duty cycle ofimage capture means that the camera is “not looking” half the time. The“backwards wagon wheels” in movies is an example of temporal aliasingdue to this normal practice of temporal undersampling. Such artifactsgenerally cannot be removed without human-assisted reconstruction. Thus,there will always be cases which cannot be automatically corrected.However, the motion conversion results available in current technologyshould be reasonable on most material.

The price of a single high definition camera or tape machine would besimilar to the cost of such a converter. Thus, in a studio havingseveral cameras and tape machines, the cost of such conversion becomesmodest. However, performing such processing adequately is presentlybeyond the budget of home and office products. Thus, the complexprocessing to remove interlace and convert the frame rate for existingmaterial is preferably accomplished at the origination studio. This isshown in FIG. 5, which is a block diagram showing 60 Hz interlaced inputfrom cameras 60 or other sources (such as non-film video tape) 62 to aconverter 64 that includes a de-interlacer function and a frame rateconversion function that can output a 36 Hz signal (36 Hz base layeronly) and a 72 Hz signal (36 Hz base layer plus 36 Hz from the temporalenhancement layer).

As an alternative to outputting a 72 Hz signal (36 Hz base layer plus 36Hz from the temporal enhancement layer), this conversion process can beadapted to produce a second MPEG-2 24 Hz temporal enhancement layer onthe 36 Hz base layer which would reproduce the original 60 Hz signal,although de-interlaced. If similar quantization is used for the 60 Hztemporal enhancement layer B frames, the data rate should be slightlyless than the 72 Hz temporal enhancement layer, since there are fewer Bframes.

-   -   >60 I 36+36=72    -   >60 I 36+24=60    -   >72 36, 72, 60    -   >50 I 36, 50, 72    -   >60 24, 36, 72

The vast majority of material of interest to the United States is lowresolution NTSC. At present, most NTSC signals are viewed withsubstantial impairment on most home televisions. Further, viewers havecome to accept the temporal impairments inherent in the use of 3-2pulldown to present film on television. Nearly all prime-time televisionis made on film at 24 frames per second. Thus, only sports, news, andother video-original shows need be processed in this fashion. Theartifacts and losses associated with converting these shows to a 36/72Hz format are likely to be offset by the improvements associated withhigh-quality de-interlacing of the signal.

Note that the motion blur inherent in the 60 Hz (or 59.94 Hz) fieldsshould be very similar to the motion blur in 72 Hz frames. Thus, thistechnique of providing a base and enhancement layer should appearsimilar to 72 Hz origination in terms of motion blur. Accordingly, fewviewers will notice the difference, except possibly as a slightimprovement, when interlaced 60 Hz NTSC material is processed into a 36Hz base layer, plus 24 Hz from the temporal enhancement layer, anddisplayed at 60 Hz. However, those who buy new 72 Hz digitalnon-interlaced televisions will notice a small improvement when viewingNTSC, and a major improvement when viewing new material captured ororiginated at 72 Hz. Even the decoded 36 Hz base layer presented on 72Hz displays will look as good as high quality digital NTSC, replacinginterlace artifacts with a slower frame rate.

The same process can also be applied to the conversion of existing PAL50 Hz material to a second MPEG-2 enhancement layer. PAL video tapes arebest slowed to 48 Hz prior to such conversion. Live PAL requiresconversion using the relatively unrelated rates of 50, 36, and 72 Hz.Such converter units presently are only affordable at the source ofbroadcast signals, and are not presently practical at each receivingdevice in the home and office.

Resolution Scalability

It is possible to enhance the base resolution template usinghierarchical resolution scalability utilizing MPEG-2 to achieve higherresolutions built upon a base layer. Use of enhancement can achieveresolutions at 1.5× and 2× the base layer. Double resolution can bebuilt in two steps, by using 3/2 then 4/3, or it can be a singlefactor-of-two step. This is shown in FIG. 7.

The process of resolution enhancement can be achieved by generating aresolution enhancement layer as an independent MPEG-2 stream andapplying MPEG-2 compression to the enhancement layer. This techniquediffers from the “spatial scalability” defined with MPEG-2, which hasproven to be highly inefficient. However, MPEG-2 contains all of thetools to construct an effective layered resolution to provide spatialscalability. The preferred layered resolution encoding process of thepresent invention is shown in FIG. 8. The preferred decoding process ofthe present invention is shown in FIG. 9.

Resolution Layer Coding. In FIG. 8, an original 2k×1k image 80 isfiltered in conventional fashion to ½ resolution in each dimension tocreate a 1024×512 base layer 81. The base layer 81 is then compressedaccording to conventional MPEG-2 algorithms, generating an MPEG-2 baselayer 82 suitable for transmission. Importantly, full MPEG-2 motioncompensation can be used during this compression step. That same signalis then decompressed using conventional MPEG-2 algorithms back to a1024×512 image 83. The 1024×512 image 83 is expanded (for example, bypixel replication, or preferably by better filters such as splineinterpolation) to a first 2k×1k enlargement 84.

Meanwhile, as an optional step, the filtered 1024×512 base layer 81 isexpanded to a second 2k×1k enlargement 85. This second 2k×1k enlargement85 is subtracted from the original 2k×1k image 80 to generate an imagethat represents the top octave of resolution between the original highresolution image 80 and the original base layer image 81. The resultingimage is optionally multiplied by a sharpness factor or weight, and thenadded to the difference between the original 2k×1k image 80 and thesecond 2k×1k enlargement 85 to generate a center-weighted 2k×1kenhancement layer source image 86. This enhancement layer source image86 is then compressed according to conventional MPEG-2 algorithms,generating a separate MPEG-2 resolution enhancement layer 87 suitablefor transmission. Importantly, full MPEG-2 motion compensation can beused during this compression step.

Resolution Layer Decoding. In FIG. 9, the base layer 82 is decompressedusing conventional MPEG-2 algorithms back to a 1024×512 image 90. The1024×512 image 90 is expanded to a first 2k×1k image 91. Meanwhile, theresolution enhancement layer 87 is decompressed using conventionalMPEG-2 algorithms back to a second 2k×1k image 92. The first 2k×1k image91 and the second 2k×1k image 92 are then added to generate ahigh-resolution 2k×1k image 93.

Improvements Over MPEG-2. In essence, the enhancement layer is createdby expanding the decoded base layer, taking the difference between theoriginal image and the decode base layer, and compressing. However, acompressed resolution enhancement layer may be optionally added to thebase layer after decoding to create a higher resolution image in thedecoder. The inventive layered resolution encoding process differs fromMPEG-2 spatial scalability in several ways:

-   -   The enhancement layer difference picture is compressed as its        own MPEG-2 data stream, with I, B, and P frames. This difference        represents the major reason that resolution scalability, as        proposed here, is effective, where MPEG-2 spatial scalability is        ineffective. The spatial scalability defined within MPEG-2        allows an upper layer to be coded as the difference between the        upper layer picture and the expanded base layer, or as a motion        compensated MPEG-2 data stream of the actual picture, or a        combination of both. However, neither of these encodings is        efficient. The difference from the base layer could be        considered as an I frame of the difference, which is inefficient        compared to a motion-compensated difference picture, as in the        present invention. The upper-layer encoding defined within        MPEG-2 is also inefficient, since it is identical to a complete        encoding of the upper layer. The motion compensated encoding of        the difference picture, as in the present invention, is        therefore substantially more efficient.    -   Since the enhancement layer is an independent MPEG-2 data        stream, the MPEG-2 systems transport layer (or another similar        mechanism) must be used to multiplex the base layer and        enhancement layer.    -   The expansion and resolution reduction filtering can be a        gaussian or spline function, which are more optimal than the        bilinear interpolation specified in MPEG-2 spatial scalability.    -   The image aspect ratio must match between the lower and higher        layers in the preferred embodiment. In MPEG-2 spatial        scalability, extensions to width and/or height are allowed. Such        extensions are not allowed in the preferred embodiment due to        efficiency requirements.    -   Due to efficiency requirements, and the extreme amounts of        compression used in the enhancement layer, the entire area of        the enhancement layer is not coded. Usually, the area excluded        from enhancement will be the border area. Thus, the 2k×1k        enhancement layer source image 86 in the preferred embodiment is        center-weighted. In the preferred embodiment, a fading function        (such as linear weighting) is used to “feather” the enhancement        layer toward the center of the image and away from the border        edge to avoid abrupt transitions in the image. Moreover, any        manual or automatic method of determining regions having detail        which the eye will follow can be utilized to select regions        which need detail, and to exclude regions where extra detail is        not required. All of the image has detail to the level of the        base layer, so all of the image is present. Only the areas of        special interest benefit from the enhancement layer. In the        absence of other criteria, the edges or borders of the frame can        be excluded from enhancement, as in the center-weighted        embodiment described above. The MPEG-2 parameters        “lower_(—)layer_(—)prediction_(—)horizontal&vertical offset”        parameters used as signed negative integers, combined with the        “horizontal&vertical_(—)subsampling_(—)factor_(—)m&n” values,        can be used to specify the enhancement layer rectangle's overall        size and placement within the expanded base layer.    -   A sharpness factor is added to the enhancement layer to offset        the loss of sharpness which occurs during quantization. Care        must be taken to utilize this parameter only to restore the        clarity and sharpness of the original picture, and not to        enhance the image. As noted above with respect to FIG. 8, the        sharpness factor is the “high octave” of resolution between the        original high resolution image 80 and the original base layer        image 81 (after expansion). This high octave image will be quite        noisy, in addition to containing the sharpness and detail of the        high octave of resolution. Adding too much of this image can        yield instability in the motion compensated encoding of the        enhancement layer. The amount that should be added depends upon        the level of the noise in the original image. A typical        weighting value is 0.25. For noisy images, no sharpness should        be added, and it even may be advisable to suppress the noise in        the original for the enhancement layer before compressing using        conventional noise suppression techniques which preserve detail.    -   Temporal and resolution scalability are intermixed by utilizing        B frames for temporal enhancement from 36 to 72 Hz in both the        base and resolution enhancement layers. In this way, four        possible levels of decoding performance are possible with two        layers of resolution scalability, due to the options available        with two levels of temporal scalability.

These differences represent substantial improvements over MPEG-2 spatialand temporal scalability. However, these differences are stillconsistent with MPEG-2 decoder chips, although additional logic may berequired in the decoder to perform the expansion and addition in theresolution enhancement decoding process shown in FIG. 9. Such additionallogic is nearly identical to that required by the less effective MPEG-2spatial scalability.

Optional Non-MPEG-2 Coding of the Resolution Enhancement Layer. It ispossible to utilize a different compression technique for the resolutionenhancement layer than MPEG-2. Further, it is not necessary to utilizethe same compression technology for the resolution enhancement layer asfor the base layer. For example, motion-compensated block wavelets canbe utilized to match and track details with great efficiency when thedifference layer is coded. Even if the most efficient position forplacement of wavelets jumps around on the screen due to changing amountsof differences, it would not be noticed in the low-amplitude enhancementlayer. Further, it is not necessary to cover the entire image—it is onlynecessary to place the wavelets on details. The wavelets can have theirplacement guided by detail regions in the image. The placement can alsobe biased away from the edge.

Multiple Resolution Enhancement Layers. At the bit rates being describedhere, where 2 MPixels (2048×1024) at 72 frames per second are beingcoded in 18.5 mbits/second, only a base layer (1024×512 at 72 fps) and asingle resolution enhancement layer have been successfully demonstrated.However, the anticipated improved efficiencies available from furtherrefinement of resolution enhancement layer coding should allow formultiple resolution enhancement layers. For example, it is conceivablethat a base layer at 512×256 could be resolution-enhanced by four layersto 1024×512, 1536×768, and 2048×1024. This is possible with existingMPEG-2 coding at the movie frame rate of 24 frames per second. At highframe rates such as 72 frames per second, MPEG-2 does not providesufficient efficiency in the coding of resolution-enhancement layers toallow this many layers at present.

Mastering Formats

Utilizing a template at or near 2048×1024 pixels, it is possible tocreate a single digital moving image master format source for a varietyof release formats. As shown in FIG. 6, a 2k×1k template can efficientlysupport the common widescreen aspect ratios of 1.85:1 and 2.35:1. A2k×1k template can also accommodate 1.33:1 and other aspect ratios.

Although integers (especially the factor of 2) and simple fractions (3/2& 4/3) are most efficient step sizes in resolution layering, it is alsopossible to use arbitrary ratios to achieve any required resolutionlayering. However, using a 2048×1024 template, or something near it,provides not only a high quality digital master format, but also canprovide many other convenient resolutions from a factor of two baselayer (1k×512), including NTSC, the U.S. television standard.

It is also possible to scan film at higher resolutions such as 4k×2k,4k×3k, or 4k×4k. Using optional resolution enhancement, these higherresolutions can be created from a central master format resolution near2k×1k. Such enhancement layers for film will consist of both imagedetail, grain, and other sources of noise (such as scanner noise).Because of this noisiness, the use of compression technology in theenhancement layer for these very high resolutions will requirealternatives to MPEG-2 types of compression. Fortunately, othercompression technologies exist which can be utilized for compressingsuch noisy signals, while still maintaining the desired detail in theimage. One example of such a compression technology is motioncompensated wavelets or motion compensated fractals.

Preferably, digital mastering formats should be created in the framerate of the film if from existing movies (i.e., at 24 frames persecond). The common use of both 3-2 pulldown and interlace would beinappropriate for digital film masters. For new digital electronicmaterial, it is hoped that the use of 60 Hz interlace will cease in thenear future, and be replaced by frame rates which are more compatiblewith computers, such as 72 Hz, as proposed here. The digital imagemasters should be made at whatever frame rate the images are captured,whether at 72 Hz, 60 Hz, 36 Hz, 37.5 Hz, 75 Hz, 50 Hz, or other rates.

The concept of a mastering format as a single digital source pictureformat for all electronic release formats differs from existingpractices, where PAL, NTSC, letterbox, pan-and-scan, HDTV, and othermasters are all generally independently made from a film original. Theuse of a mastering format allows both film and digital/electronic showsto be mastered once, for release on a variety of resolutions andformats.

Combined Resolution and Temporal Enhancement Layers

As noted above, both temporal and resolution enhancement layering can becombined. Temporal enhancement is provided by decoding B frames. Theresolution enhancement layer also has two temporal layers, and thus alsocontains B frames.

For 24 fps film, the most efficient and lowest cost decoders might useonly P frames, thereby minimizing both memory and memory bandwidth, aswell as simplifying the decoder by eliminating B frame decoding. Thus,in accordance with the present invention, decoding movies at 24 fps anddecoding advanced television at 36 fps could utilize a decoder without Bframe capability. B frames can then be utilized between P frames toyield the higher temporal layer at 72 Hz, as shown in FIG. 3, whichcould be decoded by a second decoder. This second decoder could also besimplified, since it would only have to decode B frames.

Such layering also applies to the enhanced resolution layer, which cansimilarly utilize only P and I frames for 24 and 36 fps rates. Theresolution enhancement layer can add the full temporal rate of 72 Hz athigh resolution by adding B frame decoding within the resolutionenhancement layer.

The combined resolution and temporal scalable options for a decoder areillustrated in FIG. 10. This example also shows an allocation of theproportions of an approximately 18 mbits/second data stream to achievethe spatio-temporal layered Advanced Television of the presentinvention.

In FIG. 10, a base layer MPEG-2 1024×512 pixel data stream (comprisingonly P frames in the preferred embodiment) is applied to a baseresolution decoder 100. Approximately 5 mbits/per sec of bandwidth isrequired for the P frames. The base resolution decoder 100 can decode at24 or 36 fps. The output of the base resolution decoder 100 compriseslow resolution, low frame rate images (1024×512 pixels at 24 or 36 Hz).

The B frames from the same data stream are parsed out and applied to abase resolution temporal enhancement layer decoder 102. Approximately 3mbits/per sec of bandwidth is required for such B frames. The output ofthe base resolution decoder 100 is also coupled to the temporalenhancement layer decoder 102. The temporal enhancement layer decoder102 can decode at 36 fps. The combined output of the temporalenhancement layer decoder 102 comprises low resolution, high frame rateimages (1024×512 pixels at 72 Hz).

Also in FIG. 10, a resolution enhancement layer MPEG-2 2k×1k pixel datastream (comprising only P frames in the preferred embodiment) is appliedto a base temporal high resolution enhancement layer decoder 104.Approximately 6 mbits/per sec of bandwidth is required for the P frames.The output of the base resolution decoder 100 is also coupled to thehigh resolution enhancement layer decoder 104. The high resolutionenhancement layer decoder 104 can decode at 24 or 36 fps. The output ofthe high resolution enhancement layer decoder 104 comprises highresolution, low frame rate images (2k×1k pixels at 24 or 36 Hz).

The B frames from the same data stream are parsed out and applied to ahigh resolution temporal enhancement layer decoder 106. Approximately 4mbits/per sec of bandwidth is required for such B frames. The output ofthe high resolution enhancement layer decoder 104 is coupled to the highresolution temporal enhancement layer decoder 106. The output of thetemporal enhancement layer decoder 102 is also coupled to the highresolution temporal enhancement layer decoder 106. The high resolutiontemporal enhancement layer decoder 106 can decode at 36 fps. Thecombined output of the high resolution temporal enhancement layerdecoder 106 comprises high resolution, high frame rate images (2k×1kpixels at 72 Hz).

Note that the compression ratio achieved through this scalable encodingmechanism is very high, indicating excellent compression efficiency.These ratios are shown in TABLE 5 for each of the temporal andscalability options from the example in FIG. 10. These ratios are basedupon source RGB pixels at 24 bits/pixel. (If the 16 bits/pixel ofconventional 4:2:2 encoding or the 12 bits/pixel of conventional 4:2:0encoding are factored in, then the compression ratios would be 3/4 and1/2, respectively, of the values shown.)

TABLE 5 Comp. Rate Data Rate - mb/s Ratio Layer Resolution (Hz)(typical) MPixels/s (typical) Base 1k × 512 36 5 18.9 90 Base 1k × 51272 8 (5 + 3) 37.7 113 Temp. High 2k × 1k 36 11 (5 + 6) 75.5 165 High 2k× 1k 72 18 (5 + 3 + 6 + 4) 151 201 Temp. for comparison: CCIR 601 720 ×486 29.97 5 10.5 50

These high compression ratios are enabled by two factors:

-   1) The high temporal coherence of high-frame-rate 72 Hz images;-   2) The high spatial coherence of high resolution 2k×1k images;-   3) Application of resolution detail enhancement to the important    parts of the image (e.g., the central heart), and not to the less    important parts (e.g., the borders of the frame).

These factors are exploited in the inventive layered compressiontechnique by taking advantage of the strengths of the MPEG-2 encodingsyntax. These strengths include bi-directionally interpolated B framesfor temporal scalability. The MPEG-2 syntax also provides efficientmotion representation through the use of motion-vectors in both the baseand enhancement layers. Up to some threshold of high noise and rapidimage change, MPEG-2 is also efficient at coding details instead ofnoise within an enhancement layer through motion compensation inconjunction with DCT quantization. Above this threshold, the databandwidth is best allocated to the base layer. These MPEG-2 mechanismswork together when used according to the present invention to yieldhighly efficient and effective coding which is both temporally andspatially scalable.

In comparison to 5 mbits/second encoding of CCIR 601 digital video, thecompression ratios in TABLE 5 are much higher. One reason for this isthe loss of some coherence due to interlace. Interlace negativelyaffects both the ability to predict subsequent frames and fields, aswell as the correlation between vertically adjacent pixels. Thus, amajor portion of the gain in compression efficiency described here isdue to the absence of interlace.

The large compression ratios achieved by the present invention can beconsidered from the perspective of the number of bits available to codeeach MPEG-2 macroblock. As noted above, macroblock is a 16×6 pixelgrouping of four 8×8 DCT blocks, together with one motion vector for Pframes, and one or two motion vectors for B frames. The bits availableper macroblock for each layer are shown in TABLE 6.

TABLE 6 Data Rate - mb/s Average Available Layer (typical) MPixels/sBits/Macroblock Base 5 19 68 Base Temporal 8 (5 + 3) 38 54 High 11 (5 +6) 76 37 overall, 20/ enh. layer High w/border 11 (5 + 6) 61 46 overall,35/ around hi-res enh. layer center High Temporal 18 (5 + 3 + 6 + 4) 15130 overall, 17/ enh. layer High Temporal 18 (5 + 3 + 6 + 4) 123 37overall, 30/ w/border around enh. layer hi-res center for comparison:CCIR 601 5 10.5 122

The available number of bits to code each macroblock is smaller in theenhancement layer than in the base layer. This is appropriate, since itis desirable for the base layer to have as much quality as possible. Themotion vector requires 8 bits or so, leaving 10 to 25 bits for themacroblock type codes and for the DC and AC coefficients for all four8×8 DCT blocks. This leaves room for only a few “strategic” ACcoefficients. Thus, statistically, most of the information available foreach macroblock must come from the previous frame of an enhancementlayer.

It is easily seen why the MPEG-2 spatial scalability is ineffective atthese compression ratios, since there is not sufficient data spaceavailable to code enough DC and AC coefficients to represent the highoctave of detail represented by the enhancement difference image. Thehigh octave is represented primarily in the fifth through eighthhorizontal and vertical AC coefficients. These coefficients cannot bereached if there are only a few bits available per DCT block.

The system described here gains its efficiency by utilizing motioncompensated prediction from the previous enhancement difference frame.This is demonstrably effective in providing excellent results intemporal and resolution (spatial) layered encoding.

Graceful Degradation The temporal scaling and resolution scalingtechniques described here work well for normal-running material at 72frames per second using a 2k×1k original source. These techniques alsowork well on film-based material which runs at 24 fps. At high framerates, however, when a very noise-like image is coded, or when there arenumerous shot cuts within an image stream, the enhancement layers maylose the coherence between frames which is necessary for effectivecoding. Such loss is easily detected, since thebuffer-fullness/rate-control mechanism of a typical MPEG-2encoder/decoder will attempt to set the quantizer to very coarsesettings. When this condition is encountered, all of the bits normallyused to encode the resolution enhancement layers can be allocated to thebase layer, since the base layer will need as many bits as possible inorder to code the stressful material. For example, at between about 0.5and 0.33 MPixels per frame for the base layer, at 72 frames per second,the resultant pixel rate will be 24 to 36 MPixels/second. Applying allof the available bits to the base layer provides about 0.5 to 0.67million additional bits per frame at 18.5 mbits/second, which should besufficient to code very well, even on stressful material.

Under more extreme cases, where every frame is very noise-like and/orthere are cuts happening every few frames, it is possible to gracefullydegrade even further without loss of resolution in the base layer. Thiscan be done by removing the B frames coding the temporal enhancementlayer, and thus allow use of all of the available bandwidth (bits) forthe I and P frames of the base layer at 36 fps. This increases theamount of data available for each base layer frame to between about 1.0and 1.5 mbits/frame (depending on the resolution of the base layer).This will still yield the fairly good motion rendition rate of 36 fps atthe fairly high quality resolution of the base layer, under what wouldbe extremely stressful coding conditions. However, if the base-layerquantizer is still operating at a coarse level under about 18.5mbits/second at 36 fps, then the base layer frame rate can bedynamically reduced to 24, 18, or even 12 frames per second (which wouldmake available between 1.5 and 4 mbits for every frame), which should beable to handle even the most pathological moving image types. Methodsfor changing frame rate in such circumstances are known in the art.

The current proposal for U.S. advanced television does not allow forthese methods of graceful degradation, and therefore cannot perform aswell on stressful material as the inventive system.

In most MPEG-2 encoders, the adaptive quantization level is controlledby the output buffer fullness. At the high compression ratios involvedin the resolution enhancement layer of the present invention, thismechanism may not function optimally. Various techniques can be used tooptimize the allocation of data to the most appropriate image regions.The conceptually simplest technique is to perform a pre-pass of encodingover the resolution enhancement layer to gather statistics and to searchout details which should be preserved. The results from the pre-pass canbe used to set the adaptive quantization to optimize the preservation ofdetail in the resolution enhancement layer. The settings can also beartificially biased to be non-uniform over the image, such that imagedetail is biased to allocation in the main screen regions, and away fromthe macroblocks at the extreme edges of the frame.

Except for leaving an enhancement-layer border at high frame rates, noneof these adjustments are required, since existing decoders function wellwithout such improvements. However, these further improvements areavailable with a small extra effort in the enhancement layer encoder.

Conclusion

The choice of 36 Hz as a new common ground temporal rate appears to beoptimal. Demonstrations of the use of this frame rate indicate that itprovides significant improvement over 24 Hz for both 60 Hz and 72 Hzdisplays. Images at 36 Hz can be created by utilizing every other framefrom 72 Hz image capture. This allows combining a base layer at 36 Hz(preferably using P frames) and a temporal enhancement layer at 36 Hz(using B frames) to achieve a 72 Hz display.

The “future-looking” rate of 72 Hz is not compromised by the inventiveapproach, while providing transition for 60 Hz analog NTSC display. Theinvention also allows a transition for other 60 Hz displays, if otherpassive-entertainment-only (computer incompatible) 60 Hz formats underconsideration are accepted.

Resolution scalability can be achieved though using a separate MPEG-2image data stream for a resolution enhancement layer. Resolutionscalability can take advantage of the B frame approach to providetemporal scalability in both the base resolution and enhancementresolution layers.

The invention described here achieves many highly desirable features. Ithas been claimed by some involved in the U.S. advanced televisionprocess that neither resolution nor temporal scalability can be achievedat high definition resolutions within the approximately 18.5mbits/second available in terrestrial broadcast. However, the presentinvention achieves both temporal and spatial-resolution scalabilitywithin this available data rate.

It has also been claimed that 2 MPixels at high frame rates cannot beachieved without the use of interlace within the available 18.5mbits/second data rate. However, achieves not only resolution (spatial)and temporal scalability, it can provide 2 MPixels at 72 frames persecond.

In addition to providing these capabilities, the present invention isalso very robust, particularly compared to the current proposal foradvanced television. This is made possible by the allocation of most orall of the bits to the base layer when very stressful image material isencountered. Such stressful material is by its nature both noise-likeand very rapidly changing. In these circumstances, the eye cannot seedetail associated with the enhancement layer of resolution. Since thebits are applied to the base layer, the reproduced frames aresubstantially more accurate than the currently proposed advancedtelevision system, which uses a single constant higher resolution.

Thus, the inventive system optimizes both perceptual and codingefficiency, while providing maximum visual impact. This system providesa very clean image at a resolution and frame rate performance that hadbeen considered by many to be impossible. It is believed that theinventive system is likely to outperform the advanced television formatsbeing proposed at this time. In addition to this anticipated superiorperformance, the present invention also provides the highly valuablefeatures of temporal and resolution layering.

Encryption & Watermarking

Overview

Layered compression allows a form of modularized decomposition of animage that supports flexible encryption and watermarking techniques.Using layered compression, the base layer and various internalcomponents of the base layer can be used to encrypt and/or watermark acompressed layered movie data stream. Encrypting and watermarking thecompressed data stream reduces the amount of required processingcompared to a high resolution data stream, which must be processed atthe rate of the original data. The amount of computing time required forencryption and watermarking depends on the amount of data that must beprocessed. For a particular level of computational resources, reducingthe amount of data through layered compression can yield improvedencryption strength, or reduced the cost of encryption/decryption, or acombination of both.

Encryption allows protection of the compressed image (and audio) data sothat only users with keys can easily access the information. Layeredcompression divides images into components: a temporal and spatial baselayer, plus temporal and spatial enhancement layer components. The baselayer is the key to decoding a viewable picture. Thus, only the temporaland spatial base layer need be encrypted, thereby reducing the requiredamount of computation. The enhancement layers, both temporal andspatial, are of no value without the decrypted and decompressed baselayer. Accordingly, by using such a layered subset of the bits, theentire picture stream can be made unrecognizable by encrypting only asmall fraction of the bits of the entire stream. A variety of encryptionalgorithms and strengths can be applied to various portions of thelayered stream, including enhancement layers. Encryption algorithms orkeys also can be changed as often as at each slice boundary (a datastream structure meant for signal error recovery), to provide greaterintertwining of the encryption and the picture stream.

Watermarking invisibly (or nearly invisibly) marks copies of a work. Theconcept originates with the practice of placing an identifiable symbolwithin paper to ensure that a document (e.g., money) is genuine.Watermarking allows the tracking of copies which may be removed from thepossession of an authorized owner or licensee. Thus, watermarking canhelp track lost or stolen copies back to a source, so that the nature ofthe method of theft can be determined and so that those involved in atheft can be identified.

The concept of watermarking has been applied to images, by attempting toplace a faint image symbol or signature on top of the real image beingpresented. The most widely held concept of electronic watermarking isthat it is a visible low-amplitude image, impressed on top of thevisible high-amplitude image. However, this approach alters the qualityof the original image slightly, similar to the process of impressing anetwork logo in the corner of the screen on television. Such alterationis undesirable because it reduces picture quality.

In the compressed domain, it is possible to alter signals and impresswatermark symbols or codes upon them without these watermark alterationsbeing applied directly in the visual domain. For example, the DCTtransformation operates in frequency transform space. Any alterations inthis space, especially if corrected from frame to frame, may be muchless visible (or completely invisible). Watermarking preferably uses loworder bits in certain coefficients in certain frames of a layeredcompression movie stream to provide reliable identification while beinginvisible or nearly invisible to the eye. Watermarking can be applied tothe base layer of a compressed data stream. However, it is possible toprotect enhancement layers to a much greater degree than the base layer,since the enhancement layers are very subtle in detail to begin with.Each enhancement layer can have its own unique identifying watermarkstructure.

In general, care must be taken to ensure that encryption andwatermarking are co-mingled such that the watermark cannot be easilystripped out of the stream. For this reason, it is valuable to applywatermarks in a variety of useful locations within a layered datastream. However, since the watermark is most useful in detection ofpirates and the path of the piracy, one must assume that encryption mayhave been completely or partially compromised, and thus watermarkingshould be robustly ingrained in the data stream in such a way that nosimple procedure can be applied to remove the various watermarks. Thepreferred approach is to have a secure master representation of a workand provide random variations from the master to uniquely create eachwatermark. Such random variations cannot be removed, since there is noway from the final stream to detect what the variations might have been.However, to guard against additional random variations being added tothe pirated stream to confound the watermark (perhaps by adding visiblelevels of noise to the image), it is useful to have a variety of othertechniques (such as the motion vector second-best technique describedbelow) to define watermarks.

Encryption preferably operates in such a fashion as to scramble, or atleast visibly impair, as many frames as possible from the smallestpossible units of encryption. Compression systems such as the varioustypes of MPEG and motion-compensated-wavelets utilize a hierarchy ofunits of information which must be processed in cascade in order todecode a range of frames (a “Group of Pictures,” or GOP). Thischaracteristic affords opportunities early in the range of concatenateddecoded units to encrypt in such a way as to scramble a large range offrames from a small number of parameters. Further, to protect a workcommercially, not every unit need by encrypted or confounded by theencryption of a higher-level unit. For example, a film may be renderedworthless to pirates if every other minute of film frames orparticularly important plot or action scenes are encrypted orconfounded.

In contrast, watermarking has the goal of placing a symbol and/orserial-number-style identification marks on the image stream which aredetectable to analysis, but which are invisible or nearly invisible inthe image (i.e., yielding no significant visual impairment). Thus,watermarking preferably is applied in portions of the decoding unitchain which are near the end of the hierarchy of units, to yield aminimum impact on each frame within a group of frames.

For example, FIG. 11 shows a diagram of the scope of encryption andwatermarking as a function of unit dependency with respect to I, P, andB frames. Encryption of any frame confounds all subsequent dependentframes. Thus, encryption of the first I frame confounds all P and Bframes derived from that I frame. In contrast, a watermark on that Iframe generally would not carry over to subsequent frames, and thus itis better to watermark the larger number of B frames to provide greaterprevalence of the watermark throughout the data stream.

Units of Video Information. A compressed MPEG-type ormotion-compensated-wavelets bitstream is parsed by normally extractingand processing various fundamental units of compressed information invideo. This is true of the most efficient compression systems such asMPEG-2, MPEG-4, and motion-compensated wavelets (considering wavelets tohave I, P, and B frame equivalents). Such units may consist ofmulti-frame units (such as a GOP), single frame units (e.g., I, P, and Bframe types and their motion-compensated-wavelet equivalents), sub-frameunits (such as AC and DC coefficients, macro blocks, and motionvectors), and “distributed units” (described below).

When GOPs are used as a unit of encryption, each GOP can be encryptedwith independent methods and/or keys. In this way, each GOP can have thebenefits of unique treatment and modularity, and can be decoded and/ordecrypted in parallel or out-of-order with other GOPs in non-realtime ornear-realtime (slightly delayed by a few seconds) applications (such aselectronic cinema and broadcast). The final frames need only be orderedfor final presentation.

As suggested above, encryption of certain units may confound properdecoding of other units that dependent on information derived from theencrypted unit. That is, some information within a frame may be requiredfor decoding the video information of subsequent frames; encrypting onlythe earlier frame confounds decoding of later frames that are nototherwise encrypted. Thus, in selecting units to encrypt, it is usefulto note how encryption of particular units can confound the usability ofother, related units. For example, multiple frames spanning a GOP areinfluenced at various levels as set forth in TABLE 7:

TABLE 7 Encryption of this Unit: Confounds: I frame starting a GOPentire associated GOP P frame within a GOP the remainder of the GOP Bframe within a GOP only itself as a frame

Further, an entire frame need not be encrypted to confound some or allof a GOP. Sub-units of frames may be encrypted and still have aconfounding affect, while reducing encryption and decryption processingtime. For example, encryption of the certain intra-frame unitsinfluences subsequent frames at various levels as set forth in TABLE 8:

TABLE 8 Encryption of this Unit: Confounds: DC coefficients in an Iframe (the top all frames in the associated GOP hierarchy unit of DCcoefficients) DC coefficients in a P frame the remainder of theassociated GOP DC Coefficients in a B frame (the only that frame bottomhierarchy unit of DC coefficients) motion vectors in a P frame theremainder of the associated GOP motion vectors in a B frame only thatframe AC coefficients in an I frame (top the remainder of the associatedhierarchy unit of AC coefficients) GOP AC coefficients in a P frame(bottom the remainder of the associated hierarchy unit of ACcoefficients) GOP macroblock mode bits in a P frame the remainder of theassociated (e.g., the macroblock modes “forward GOP predict”, “intra”,and “4MV”) macroblock mode bits in a B frame only that frame (“forward”,“backward”, “bi- directional”, “direct” mode bits in MPEG-4) “sliceboundaries” (usually left-edge only the following slice beginnings ofmacroblock lines, where various parameters are reset) left column ofmacroblocks for each that P frame and subsequent P P and B frame (e.g.,motion vectors frames for P; the corresponding B are reset at the leftcolumn, with each frame for B macroblock to the right beingdifferentially determined from the left (and possibly above in MPEG-4))base layer in a resolution itself and all higher resolution enhancementlayer system (top layers hierarchy unit in resolution) enhancement layerin resolution itself and all higher resolution (bottom and lower unitsof resolution) layers base temporal layer (top hierarchy itself and allhigher temporal unit) layers temporal enhancement layer(s) its ownframes (B frames, lower temporal hierarchy units)

Delay can be applied in many applications (such as broadcast and digitalcinema), allowing an aggregation of items from units of like types to beencrypted before transmission. This allows for a “distributed unit”,where the bits comprising an encryption/decryption unit are physicallyallocated across a data stream in conventional units of the typedescribed above, making decrypting without knowledge of the key evenmore difficult. For decryption, a sufficient number of conventionalunits would be aggregated (e.g., in a buffer) and decrypted as a group.For example, DC coefficients can be collected into groups for an entireframe or GOP. Similarly, motion vectors are coded differentially andpredicted one to the next from one macroblock to the next throughout theframe, and thus can be encrypted and decrypted in aggregations.Variable-length-coding tables can also be aggregated into groups andform modular units between “start codes”. Additional examples of unitsor subunits that can be aggregated, encrypted, and then have theencrypted bits separated or spread in the data stream include: motionvectors, DC coefficients, AC coefficients, and quantizer scale factors.

Application of Encryption

In the preferred embodiment, one or more of the units described above(or other data stream units with similar properties) may be selected forencryption, and each unit can be encrypted independently rather than asa combined stream (as with MPEG-1, MPEG-2, and MPEG-4). Encryption ofeach unit may use different keys of different strengths (e.g., number ofbits per key) and may use different encryption algorithms.

Encryption can be applied uniquely to each distinct copy of a work (whenphysical media is used, such as DVD-RAM), so that each copy has its ownkey(s). Alternatively, an encryption algorithm can be applied on theassembled stream with critical portions of the stream removed from thedata stream or altered before encryption (e.g., by setting all motionvectors for the left-hand macroblocks to zero), thus defining a bulkdistribution copy. The removed or altered portion can then be encryptedseparately and uniquely for each display site, thereby defining a customdistribution copy that is sent separately to individual sites in aconvenient manner (e.g., satellite transmission, modem, Internet, etc.).This technique is useful, for example, where the bulk of a work isdistributed on a medium such as a DVD-ROM, while unique copies of thesmaller critical compression units are separately sent, each with theirown unique keys, to independent recipient destinations (e.g., bysatellite, Internet, modem, express delivery, etc.). Only when thecustom portion is decrypted and recombined with the decrypted bulkdistribution copy will the entire work be decodable as a video signal.The larger the bandwidth (size capacity) of such custom information, thelarger the portion of the image that can be custom encrypted. Thistechnique can be used with watermarking as well.

A variant of this approach is to encrypt a subset of units from a datastream as a custom distribution copy, and not encrypt the remainingunits at all. The remaining units may be distributed in bulk form,separately from the custom distribution copy. Only when the customportion is decrypted and recombined with the unencrypted bulkdistribution copy will the entire work be decodable as a video signal.

One or more overall encryptions can be concatenated or combined withspecial customized encryptions for various of the crucial units of videodecoding information. For example, the entire video data stream may be“lightly” encrypted (e.g., using a short key or simple algorithm) whilecertain key units of the data stream are more “heavily” encrypted (e.g.,using a longer key or more complex algorithm). For example, in oneembodiment, the highest resolution and/or temporal layers may be moreheavily encrypted to define a premium signal that provides the bestappearing image when properly decrypted. Lower layers of the image wouldbe unaffected by such encryption. This approach would allow differentgrades of signal service for end-users.

If units are encrypted independently of each other, then decryption maybe performed in parallel using one or more concurrently processeddecryption methods on separate units within the compressed image stream.

Application of Watermarking

With respect to the units discussed above, and other units havingsimilar properties, various points within a compressed video data streamare suitable for applying watermarks in various ways, including:

-   -   In transform space or realspace or combinations thereof.    -   In the least significant bits (LSBs) of the DC coefficients. For        example, the DC coefficients can have extra bits (10 and 11 bits        are allowed in MPEG2, and up to 14 bits in MPEG4). The low order        bit(s) can code a specific watermark identifier without        degrading the image in any visible way. Further, these low order        bits might only be present in I frames, since a clear watermark        need not be present on every frame.    -   In noise patterns within the LSBs of the AC coefficients.    -   In low amplitude overall picture low frequencies, coded        one-frame-to-the-next, forming a visually undetectable imaged        pattern. For example, this might be a small number of low signal        amplitude letters or numbers on each frame, where each letter is        very large and soft. For example, where a pixel should have a        binary value of “84”, the watermark process could instead set        the value to “83”; the watermark at this location this has a        value of “1”. The difference is essentially invisible to the        eye, but forms a code in the compressed data stream. Such an        imaged pattern would be detected by subtracting the decoded        image from the unperturbed (unwatermarked) decompressed original        (and also from the uncompressed original source work), and then        greatly increasing the amplitude. A series of very large blurry        letters or numbers would then appear.    -   In frames which do not propagate (such as I frames, the last P        frame before an I frame, and B frames), using marks of extremely        low visibility. These frames also are displayed only briefly.    -   At slice boundaries (usually left-edge beginnings of macroblock        lines).

Watermarks at these points generally comprise imposed patterns of minorpixel data variations. In some cases, these variations form images orsymbols that are invisible or nearly invisible to the eye due to thevery low amplitude of the bit variations in terms of pixel brightnessand color and/or due to the brevity of display. For example, FIGS. 12Aand 12B show diagrams of image frames 1200 with different types ofwatermarks. FIG. 12A shows a frame 1200 with a single symbol (“X”) 1202in one corner. FIG. 12B shows a frame 1200 with a set of marks (dots, inthis example) 1204 scattered around the frame 1200. Such watermarks aredetectable only by data comparison to yield the watermark signal. Forexample, a precise decoder can detect LSB variations between an originalwork and a watermarked work that are invisible to the eye, but whichuniquely watermark the customized copy of the original work.

Other forms of watermarking may be used that do not impose specificimages or symbols, but do form unique patterns in the data streams. Forexample, certain decisions of coding are nearly invisible, and may beused to watermark a data stream. For example, minor rate controlvariations are invisible to the eye, but can be used to mark each copysuch that each copy has a slightly different number of AC coefficientsin some locations. Examples of other such decisions include:

-   -   Rate control variations within an I frame.    -   Rate control variations within P and B frames.    -   Specific AC coefficient allocations, affecting LSBs.

Similarly, second-best choices for motion vectors which are nearly asgood as optimum motion vectors may be used to create a watermark code.Also, a system can use second-best selections for exactly the same SADs(sum of absolute differences, a common motion vector match criteria)when and where they occur. Other non-optimum (e.g., third and higherranked) motion vector matches can also be used, if needed, with verylittle visual impairment. Such second-choice (and higher) motion vectorsneed only be used occasionally (e.g., a few per frame) in a coherentpattern to form a watermark code.

Image variations are less visible near the periphery of the frame (i.e.,near the top, bottom, right, edge, and left edge). It is thereforebetter to apply image or symbol type watermarks to image edge regions ifthe selected watermark is possibly slightly visible. Watermark methodsof very low visibility (such as second-best motion vectors or ratecontrol variations) can be used everywhere on the image.

Watermarking also can be coded as a unique serial-number-style code foreach watermarked copy. Thus, 1,000 copies of an original work would eachbe watermarked in a slightly different fashion using one or moretechniques described above. By tracking where each watermarked copy isshipped, it is possible to determine which copy was the source ofunauthorized duplication when a watermark is found in an unauthorizedcopy.

Watermark Detection

Most of these methods for watermarking require that the originaldecompressed image be used as a reference for comparison with eachwatermarked copy in order to reveal (decipher) the watermark. Thedifferences between the two images will disclose the watermark. Thus, itis necessary to keep the master decompressed source in a secure place.Security is required because possession of a copy of the masterdecompressed source provides sufficient information with which to defeatmany of the watermarking methods. However, theft of the watermarkingcomparison master is itself detectable, since the master isautomatically “watermarked” to be a perfect match to itself. When themaster is used to confound a copy (i.e., find and remove the watermark),it implies possession of a master.

Use of low amplitude, large blurry symbols or images as a watermark hasthe advantage that such symbols or images are detectable not only bycomparison against the decompressed master source, but also bycomparison against the uncompressed original work. Thus, an originaluncompressed work can be stored in an independent secure environment,such that low-amplitude watermarks can be used within the original(otherwise unvaried) compressed master source. In this way, a watermarkcomparison reference would remain if either the original work or thecompressed/decompressed master source are stolen. However, possession ofboth would allow defeat of both classes of watermarks.

Watermark Vulnerability

Important to the use of watermarking is an understanding of methods thatmight be used to defeat or confound the detection of such marks. Somewatermark methods are subject to confounding by adding small amounts ofnoise to the image. While this may degrade the image quality somewhat,the degradation might still be visually small, but sufficient toconfound deciphering of the watermark. Watermark techniques which arevulnerable to being confounded by adding noise include use of LSBs in DCor AC coefficients.

Other watermark methods are much more difficult to confound using noise.Those watermark techniques which are resistant to confounding by noise,but which can still be readily detected, include low amplitude overallpicture low frequency image variations (such as a low-amplitude, veryblurry large word superimposed on the image), second-best motionvectors, and minor rate control variations.

It is thus valuable to utilize multiple methods of watermarking in orderto defeat simple methods which attempt to confound the detection of thewatermark. Further, use of encryption ensures that watermarks cannot bealtered unless the encryption is compromised. Accordingly, watermarkingpreferably is used in conjunction with encryption of a suitable strengthfor the application.

Tool-Kit Approach

The various concepts of encryption and watermarking comprising thisaspect of the invention are preferably embodied as a set of tools whichcan be applied to the task of protecting valuable audio/video media. Thetools can be combined by a content developer or distributor in variousways, as desired, to create a protection system for a layered compresseddata stream.

For example, FIG. 13 is a flowchart showing one method of applying theencryption techniques of the invention. A unit to be encrypted isselected (STEP 1300). This may be any of the units described above(e.g., a distributed unit, a multi-frame unit, a single frame unit, or asub-frame unit), or other units with similar properties. An encryptionalgorithm is selected (STEP 1302). This may be a single algorithmapplied throughout an encryption session, or may be a selection perunit, as noted above. Suitable algorithms are well known, and include,for example, both private and public key algorithms, such as DES, TripleDES, RSA, Blowfish, etc. Next, one or more keys are generated (STEP1304). This involves selection of both key length and key value. Again,this may be a single selection applied throughout an encryption session,or truly may be a selection per unit, as noted above. Lastly, the unitis encrypted using the selected algorithm and key(s) (STEP 1306). Theprocess then repeats for a next unit. Of course, a number of the stepsmay be carried out in different orders, particularly steps 1300, 1302,and 1304.

For decompression, the relevant key(s) would be applied to decrypt thedata stream. Thereafter, the data stream would be decompressed anddecoded, as described above, to generate a displayable image.

FIG. 14 is a flowchart showing one method of applying the watermarkingtechniques of the invention. A unit to be watermarked is selected (STEP1400). Again, this may be any of the units described above (e.g., adistributed unit, a multi-frame unit, a single frame unit, or asub-frame unit), or other units with similar properties. One or morewatermarking techniques are then selected, such as a noise-tolerantmethod and a non-noise tolerant method (STEP 1402). This may be a singleselection applied throughout a watermarking session, or truly may be aselection per unit (or class of units, where two or more watermarkingtechniques are applied to different types of units). Lastly, theselected unit is watermarked using the selected technique (STEP 1404).The process then repeats for a next unit. Of course, a number of thesteps may be carried out in different orders, particularly steps 1400and 1402. Further,

Key Management

Encryption/decryption keys may be tied to various items of information,in order to construct more secure or synchronized keys. For example,public or private encryption and decryption keys may be generated toinclude or be derived from any of the following components:

-   -   Previous keys.    -   A serial number of a destination device (e.g., a theater        projector having a secure serial number).    -   A date or time range (using a secure clock), such that the key        only works during specific time periods (e.g., only on certain        days of the week, or only for a relative period, such as one        week). For example, an encryption system may plan for the use of        a secure GPS (global positioning satellite) in the decoder as a        source for time. The decrypting processor would only need access        to that secure time source to decrypt the image file or stream.    -   Location of the decryption processor. A GPS capability would        allow fairly exact real-time location information to be        incorporated into a key. An internet protocol (IP) static        address of a known destination could also be used.    -   Accounting records of the number of previous showings of a work,        as reported (manually or automatically) by each theater.    -   A “PIN” (personal identification number) of a specific        authorizing person (e.g., a theater manager).    -   Physical customized-encrypted movies (such as DVD's, where each        is uniquely keyed to a specific movie theater) can be used such        that the possession of the encrypted movie itself by a key        holder at the intended site is a form of key authorization for a        subsequent movie. For example, playback of a portion of the        movie and transmission of that portion to a remote key        generation site can be part of the key authorization protocol.        Further, the use of the encrypted movie data as a key element        can be tied to a secure media erasure key when a distribution        copy is stored on an erasable media, such as hard disk or        DVD-RAM. In this way, the previous movie is erased as part of        the key process for obtaining the new movie.    -   Keys can also be active for a specific number of showings or        other natural units of use, requiring new keys subsequently.

Various methods of managing distribution of keys for decryption can beapplied. Different key management strategies can be applied to each typeof use and each type of data delivery (whether network data transfer,satellite, or physical disk or tape media). Following are examples ofkey distribution and management procedures:

-   -   Keys can be stored on a media (e.g., floppy disk, CDROM) and        physically shipped to a destination via overnight shipping, or        transmitted electronically or in text format (e.g., by        facsimile, email, direct-connect data transmission, Internet        transmission, etc.).    -   Public key methods can also be used with local unique keys, as        well as authenticated third-party key verification.    -   Keys may be themselves encrypted and electronically transmitted        (e.g., via direct-connect data transmission, Internet        transmission, email, etc.), with pre-defined rules at each        destination (e.g., theater) for how to decrypt and apply the        keys.    -   Possession of a current key may be required as a condition of        obtaining or utilizing new keys. The current key value may be        transmitted to a key management site by any suitable means, as        noted above; the new key can be returned by one of the means        noted above.    -   Use of a decryption key may require a “key handshake” with a key        management site that validates or authorizes application of the        key for every instance of decryption. For example, a decryption        key may need to be combined with additional symbols maintained        by the key management site, where the specific symbols vary from        use to use. Use of key handshakes can be used for every showing,        or for every length of time of use, or for other natural value        units. Since such uses may also be a natural unit of accounting,        key management can also be integrally tied to accounting systems        which log uses or use durations, and apply appropriate charges        to the key holder (e.g., rental charges per showing for a        theater). For example, both key management and use logging can        be tied to a key authorization server system which can        simultaneously handle the accounting for each authorized showing        or use duration.

Some keys may be pre-authorized keys versus keys which are authorizedonsite. Pre-authorized keys generally would be issued one at a time by akey management site. For onsite key authorization, a key management sitemay issue a set of keys to a theater, thus allowing a local manager toauthorize additional decryptions (and hence showings) of a movie whichis more popular than originally projected, to accommodate audiencedemand). If such keys are used, the system is preferably designed tosignal (e.g., by email or data record sent over the Internet or bymodem) the key management site about the additional showings, foraccounting purposes.

CONCLUSION

Different aspects of the invention that are considered to be novelinclude (without limitation) the following concepts:

-   -   Encryption applied to layered compression    -   Watermarking applied to layered compression    -   Unique encryption applied to each layer in a layered system,        requiring different keys, authorizations, or algorithms to        unlock each independent layer    -   Unique watermarking applied to each layer in order to identify        the particular layer (using a method such as a serial number)    -   Utilizing sub-frame units of compressed image streams for        encryption or watermarking    -   Utilizing multiple simultaneous watermark methods in order to        protect against methods which attempt to confound the detection        of a particular type of watermark    -   Utilizing multiple simultaneous encryption methods and        strengths, thus requiring multiple independent decryption        systems in order to decode the various units within the        single-layer or layered compressed image stream    -   Parallel decryption using one or more simultaneous decryption        methods on various units within the compressed image stream    -   Tying keys to accounting systems    -   Tying encryption to specific media and/or a specific target        location or serial number    -   Tying encryption to a secure clock and date range of use    -   Tying encryption to a specific number of uses with a secure use        counter    -   Using the movie itself as a key to obtain new movies or keys    -   Erasure of the movie data on physical media when used as a key        to obtain new movies, or when the duration of authorized use        expires    -   Use of a flexible key toolkit approach, so that key use methods        can be continuously refined in order to improve flexibility,        convenience of use, and security    -   Use second-best (or third, etc., best) motion vectors as a        watermark technique    -   Use of minor rate control variations as a watermark technique        (applied to any combination of I, B, and/or P-type frames, as        well as their motion-compensated-wavelet equivalents)    -   Use of low-order bit variations in DC and/or AC coefficients as        a watermark technique (applied to I, B, and/or P type frames,        and their equivalents).    -   Use of low amplitude blurry letters or numbers uniquely added to        each copy of the image during compression to uniquely watermark        each copy    -   Applying encryption to portions of the bitstream which affect        large portions of the image stream (high influence for        encryption)    -   Applying overall encryption for the bulk of a work, plus        customized encryption(s) for selected units    -   Encrypting small portions of the data stream, and sending these        by point-to-point methods to each specific location (including        tying to serial numbers, keys, personnel codes, IP addresses,        and other unique identifiers at that specific location)    -   Applying watermarking to portions of the bitstream which have        low influence on other frames, to minimize visibility    -   Use of image edge regions (near top, bottom, left edge, and        right edge) for potentially visible watermarks (such as        low-amplitude letters and numbers or LSBs in DC or AC        coefficients), to minimize visual impact    -   Extraction of sub-frame unit influence points for independent        encryption, such as left column (slice start) motion vectors, DC        and AC Coefficients in I frames, prediction mode bits, control        codes, etc.

Computer Implementation

The invention may be implemented in hardware (e.g., an integratedcircuit) or software, or a combination of both. However, preferably, theinvention is implemented in computer programs executing on one or moreprogrammable computers each comprising at least a processor, a datastorage system (including volatile and non-volatile memory and/orstorage elements), an input device, and an output device. Program codeis applied to input data to perform the functions described herein andgenerate output information. The output information is applied to one ormore output devices, in known fashion.

Each such program may be implemented in any desired computer language(including machine, assembly, or high level procedural, logical, orobject oriented programming languages) to communicate with a computersystem. In any case, the language may be a compiled or interpretedlanguage.

Each such computer program is preferably stored on a storage media ordevice (e.g., ROM, CD-ROM, or magnetic or optical media) readable by ageneral or special purpose programmable computer system, for configuringand operating the computer when the storage media or device is read bythe computer system to perform the procedures described herein. Theinventive system may also be considered to be implemented as acomputer-readable storage medium, configured with a computer program,where the storage medium so configured causes a computer system tooperate in a specific and predefined manner to perform the functionsdescribed herein.

A number of embodiments of the present invention have been described.Nevertheless, it will be understood that various modifications may bemade without departing from the spirit and scope of the invention. Forexample, while the preferred embodiment uses MPEG-2 coding and decoding,the invention will work with any comparable standard that providesequivalents of I, B, and P frames and layers. Accordingly, it is to beunderstood that the invention is not to be limited by the specificillustrated embodiment, but only by the scope of the appended claims.

1. A method for encrypting a data stream of video information encodedand compressed into a base layer and at least one enhancement layer,including the steps of: (a) selecting at least one encryption algorithm;(b) selecting at least one unit of one of the base layer or at least oneenhancement layer to encrypt; (c) applying at least one selectedencryption algorithm to encrypt each selected unit into an encryptedunit; and (d) applying a first unique selected encryption algorithm toselected units from the base layer and a second unique selectedencryption algorithm to selected units from the at least one enhancementlayer.
 2. The method of claim 1, further comprising selecting units toencrypt to confound decompression or decoding of subsequent units thatdepend on information in the units to encrypt.
 3. The method of claim 1,wherein at least one selected unit is a multi-frame unit.
 4. The methodof claim 1, wherein at least one selected unit is a frame unit.
 5. Themethod of claim 1, wherein at least one selected unit is a sub-frameunit.
 6. The method of claim 1, wherein at least one selected unit is adistributed unit.
 7. The method of claim 1, further comprisingdecrypting each encrypted unit.
 8. The method of claim 1, furthercomprising decrypting a plurality of encrypted units in parallel.
 9. Amethod for encrypting a data stream of video information encoded andcompressed into a base layer and at least one enhancement layer,including the steps of: (a) selecting at least one encryption algorithm;(b) selecting at least one unit of one of the base layer or at least oneenhancement layer to encrypt; and (c) applying at least one selectedencryption algorithm to encrypt each selected unit into an encryptedunit, wherein the at least one selected encryption algorithm differsbetween at least two selected units.
 10. A method for encrypting a datastream of video information encoded and compressed into a base layer andat least one enhancement layer, including the steps of: (a) selecting atleast one encryption algorithm; (b) selecting at least one unit of oneof the base layer or at least one enhancement layer to encrypt; (c)applying at least one selected encryption algorithm to encrypt eachselected unit into an encrypted unit, wherein each selected encryptionalgorithm has a key having a key length; and (d) varying at least one ofa value for the key and the key length for at least some of the selectedunits.
 11. A method for encrypting a data stream of video informationencoded and compressed into a base layer and at least one enhancementlayer, including the steps of: (a) selecting at least one encryptionalgorithm; (b) selecting at least one unit of one of the base layer orat least one enhancement layer to encrypt; (c) applying at least oneselected encryption algorithm to encrypt each selected unit into anencrypted unit; (d) encrypting a subset of selected units from the datastream of video information to create an encrypted custom distributioncopy; (e) grouping remaining units from the data stream of videoinformation to create a bulk distribution copy, wherein the remainingunits comprise units from the data stream of video information that aredifferent from the subset of selected units; and (f) distributing thebulk distribution copy separately from the encrypted custom distributioncopy.
 12. A method for encrypting a data stream of video informationencoded and compressed into a base layer and at least one enhancementlayer, including the steps of: (a) selecting at least one encryptionalgorithm; (b) selecting at least one unit of one of the base layer orat least one enhancement layer to encrypt; and (c) applying at least oneselected encryption algorithm to encrypt each selected unit into anencrypted unit, wherein each selected encryption algorithm has a keygenerated from at least one or more of the following factors: a previouskey; a serial number of a destination decoding device; a date or timerange determined by a secure clock; a location identifier; a number ofprevious uses of a work; a PIN of a specific authorizing person; orportions of a previously encrypted data stream of video information. 13.A method for encrypting a data stream of video information encoded andcompressed into a base layer and at least one enhancement layer,including the steps of: (a) selecting at least one encryption algorithm;(b) selecting at least one unit of one of the base layer or at least oneenhancement layer to encrypt; and (c) applying at least one selectedencryption algorithm to encrypt each selected unit into an encryptedunit, wherein each selected encryption algorithm has a key; (d) storinga plurality of encrypted units on an erasable media; and (e) erasing theencrypted units upon an expiration of the keys.
 14. A system forencrypting a data stream of video information encoded and compressedinto a base layer and at least one enhancement layer, including: meansfor selecting at least one encryption algorithm; means for selecting atleast one unit of one of the base layer or at least one enhancementlayer to encrypt; means for applying at least one selected encryptionalgorithm to encrypt each selected unit into an encrypted unit; andmeans for applying a first unique selected encryption algorithm toselected units from the base layer and a second unique selectedencryption algorithm to selected units from the at least one enhancementlayer.
 15. The system of claim 14, further comprising means forselecting units to encrypt to confound decompression or decoding ofsubsequent units that depend on information in the units to encrypt. 16.The system of claim 14, wherein at least one selected unit is amulti-frame unit.
 17. The system of claim 14, wherein at least oneselected unit is a frame unit.
 18. The system of claim 14, wherein atleast one selected unit is a sub-frame unit.
 19. The system of claim 14,wherein at least one selected unit is a distributed unit.
 20. The systemof claim 14, further comprising means for decrypting each encryptedunit.
 21. The system of claim 14, further comprising means fordecrypting a plurality of encrypted units in parallel.
 22. A system forencrypting a data stream of video information encoded and compressedinto a base layer and at least one enhancement layer, including: meansfor selecting at least one encryption algorithm; means for selecting atleast one unit of one of the base layer or at least one enhancementlayer to encrypt; and means for applying at least one selectedencryption algorithm to encrypt each selected unit into an encryptedunit, wherein the at least one selected encryption algorithm differsbetween at least two selected units.
 23. A system for encrypting a datastream of video information encoded and compressed into a base layer andat least one enhancement layer, including: means for selecting at leastone encryption algorithm; means for selecting at least one unit of oneof the base layer or at least one enhancement layer to encrypt; meansfor applying at least one selected encryption algorithm to encrypt eachselected unit into an encrypted unit, wherein each selected encryptionalgorithm has a key having a key length; and means for varying at leastone of a value for the key and the key length for at least some of theselected units.
 24. A system for encrypting a data stream of videoinformation encoded and compressed into a base layer and at least oneenhancement layer, including: means for selecting at least oneencryption algorithm; means for selecting at least one unit of one ofthe base layer or at least one enhancement layer to encrypt; means forapplying at least one selected encryption algorithm to encrypt eachselected unit into an encrypted unit; means for encrypting a subset ofselected units from the data stream of video information to create anencrypted custom distribution copy; means for grouping remaining unitsfrom the data stream of video information to create a bulk distributioncopy, wherein the remaining units comprise units from the data stream ofvideo information that are different from the subset of selected units;and means for distributing the bulk distribution copy separately fromthe encrypted custom distribution copy.
 25. A system for encrypting adata stream of video information encoded and compressed into a baselayer and at least one enhancement layer, including: means for selectingat least one encryption algorithm; means for selecting at least one unitof one of the base layer or at least one enhancement layer to encrypt;and means for applying at least one selected encryption algorithm toencrypt each selected unit into an encrypted unit, wherein each selectedencryption algorithm has a key generated from at least one or more ofthe following factors: a previous key; a serial number of a destinationdecoding device; a date or time range determined by a secure clock; alocation identifier; a number of previous uses of a work; a PIN of aspecific authorizing person; or portions of a previously encrypted datastream of video information.
 26. A system for encrypting a data streamof video information encoded and compressed into a base layer and atleast one enhancement layer, including: means for selecting at least oneencryption algorithm; means for selecting at least one unit of one ofthe base layer or at least one enhancement layer to encrypt; means forapplying at least one selected encryption algorithm to encrypt eachselected unit into an encrypted unit, wherein each selected encryptionalgorithm has a key; means for storing a plurality of encrypted units onan erasable media; and means for erasing the encrypted units upon anexpiration of the keys.
 27. A computer program, stored on acomputer-readable medium, for encrypting a data stream of videoinformation encoded and compressed into a base layer and at least oneenhancement layer, the computer program comprising instructions forcausing a computer to: select at least one encryption algorithm; selectat least one unit of one of the base layer or at least one enhancementlayer to encrypt; apply at least one selected encryption algorithm toencrypt each selected unit into an encrypted unit; and apply a firstunique selected encryption algorithm to selected units from the baselayer and a second unique selected encryption algorithm to selectedunits from the at least one enhancement layer.
 28. The computer programof claim 27, further including instructions for causing the computer toselect units to encrypt to confound decompression or decoding ofsubsequent units that depend on information in the units to encrypt. 29.The computer program of claim 27, wherein at least one selected unit isa multi-frame unit.
 30. The computer program of claim 27, wherein atleast one selected unit is a frame unit.
 31. The computer program ofclaim 27, wherein at least one selected unit is a sub-frame unit. 32.The computer program of claim 27, wherein at least one selected unit isa distributed unit.
 33. The computer program of claim 27, furthercomprising instructions for causing the computer to decrypt eachencrypted unit.
 34. The computer program of claim 27, further comprisinginstructions for causing the computer to decrypt a plurality ofencrypted units in parallel.
 35. A computer program, stored on acomputer-readable medium, for encrypting a data stream of videoinformation encoded and compressed into a base layer and at least oneenhancement layer, the computer program comprising instructions forcausing a computer to: select at least one encryption algorithm; selectat least one unit of one of the base layer or at least one enhancementlayer to encrypt; and apply at least one selected encryption algorithmto encrypt each selected unit into an encrypted unit, wherein the atleast one selected encryption algorithm differs between at least twoselected units.
 36. A computer program, stored on a computer-readablemedium, for encrypting a data stream of video information encoded andcompressed into a base layer and at least one enhancement layer, thecomputer program comprising instructions for causing a computer to:select at least one encryption algorithm; select at least one unit ofone of the base layer or at least one enhancement layer to encrypt;apply at least one selected encryption algorithm to encrypt eachselected unit into an encrypted unit, wherein each selected encryptionalgorithm has a key having a key length; and vary at least one of avalue for the key and the key length for at least some of the selectedunits.
 37. A computer program, stored on a computer-readable medium, forencrypting a data stream of video information encoded and compressedinto a base layer and at least one enhancement layer, the computerprogram comprising instructions for causing a computer to: select atleast one encryption algorithm; select at least one unit of one of thebase layer or at least one enhancement layer to encrypt; apply at leastone selected encryption algorithm to encrypt each selected unit into anencrypted unit; encrypt a subset of selected units from the data streamof video information to create an encrypted custom distribution copy;group remaining units from the data stream of video information tocreate a bulk distribution copy, wherein the remaining units compriseunits from the data stream of video information that are different fromthe subset of selected units; and distribute the bulk distribution copyseparately from the encrypted custom distribution copy.
 38. A computerprogram, stored on a computer-readable medium, for encrypting a datastream of video information encoded and compressed into a base layer andat least one enhancement layer, the computer program comprisinginstructions for causing a computer to: select at least one encryptionalgorithm; select at least one unit of one of the base layer or at leastone enhancement layer to encrypt; and apply at least one selectedencryption algorithm to encrypt each selected unit into an encryptedunit, wherein each selected encryption algorithm has a key generatedfrom at least one or more of the following factors: a previous key; aserial number of a destination decoding device; a date or time rangedetermined by a secure clock; a location identifier; a number ofprevious uses of a work; a FIN of a specific authorizing person; orportions of a previously encrypted data stream of video information. 39.A computer program, stored on a computer-readable medium, for encryptinga data stream of video information encoded and compressed into a baselayer and at least one enhancement layer, the computer programcomprising instructions for causing a computer to: select at least oneencryption algorithm; select at least one unit of one of the base layeror at least one enhancement layer to encrypt; apply at least oneselected encryption algorithm to encrypt each selected unit into anencrypted unit, wherein each selected encryption algorithm has a key;store a plurality of encrypted units on an erasable media; erase theencrypted units upon an expiration of the keys.